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Introduction 

A teacher has more impact on student learning than any other 
factor controlled by school systems, including class size, 
school size and the quality of after-school programs — or even 
which school a student is attending — but currently, there is 
no agreement among education stakeholders about how to 
identify and measure effective teaching. In an effort to 
improve the quality of information about teaching 
effectiveness, in the fall of 2009, the Bill & Melinda Gates 
Foundation launched the two-year Measures of Effective 
Teaching (MET) project to rigorously develop and test multiple 
measures of teacher effectiveness. 

As part of the project, partners from more than a dozen 
reputable academic, non-profit and for-profit organizations are 
collecting and analyzing data collected during the 2009-10 
and 2010-11 school years from over 3,000 teacher volunteers 
and their classrooms across Charlotte-Mecklenburg Schools, 
Dallas Independent School District, Denver Public Schools, 
Hillsborough County Public Schools, Memphis City Schools 
and the New York City Department of Education. Teachers 
and classrooms in Pittsburgh Public Schools are also 
participating in the project by helping researchers with early- 
stage development and testing of the effectiveness measures 
before they are tested in the other MET project districts. 

The project’s data is collected across five critical research 
areas: 


1 . Student achievement gains on state standardized 
assessments and supplemental assessments designed 
to measure higher-order conceptual thinking 

2. Classroom observations and teacher reflections 

3. Teachers’ pedagogical content knowledge 

4. Student perceptions of the classroom instructional 
environment 

5. Teachers’ perceptions of working conditions and 
instructional support at their schools 

A close analysis of each of these will help establish which 
teaching practices, skills, and knowledge positively impact 
student learning. This paper seeks to define and explain how 
classroom observations and teacher reflections are being 
used to develop the “validation engine” for reviewing other 
observational protocols. 

Using Classroom Observations to 
Validate Protocols 

A teacher’s classroom instruction style is perhaps one of the 
most important and least well-understood factors contributing 
to teacher effectiveness. Data about the instruction style of 
teachers participating in the MET project is being collected 
through video observation of classroom lessons. Using video 
rather than sending observers directly into the classroom 
creates a record of practice that can be analyzed by multiple 
raters using different observation frameworks, and can also 
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2 The videotaped lessons used in the MET project research are strictly 
confidential and, for the purposes of the MET project research, will only be 
accessed by the project’s researchers and by the teacher who has been 
videotaped. The videos captured for the project will not be available for viewing 
by superintendents, principals, other teachers or district officials. The only 
videos used in the validation engine are those where the teacher has explicitly 
consented for this additional use. 
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be observed by the teachers themselves. It is also less 
intrusive for the teacher being observed. 

The MET project is using seven different observation 
protocols to determine the link between specific teaching 
practices and improved student achievement. However, there 
are many more protocols developed by researchers, states 
and districts that were not used in the study. MET project 
researchers are creating a tool called a validation engine that 
will provide an opportunity for districts to test whether their 
observation protocols (either homegrown or adapted from 
popular protocols) routinely award their highest ratings to 
teachers whose students’ academic performance grew most 
and their lowest ratings to teachers whose students’ academic 
performance grew least. 

About the Validation Engine 

The validation engine is a web-delivered software tool that 
allows users to view classroom observation videos, rate those 
videos and then receive a report that evaluates the predictive 
validity and rater consistency for the protocol. The validation 
engine, which is being developed by Empirical Education Inc, 
an education research company, has four components: 

• A secure video viewer that provides web-based access 
to a video library of teacher lessons for raters to watch. 

• A score sheet template that allows raters to enter 
scores for each observed video into a web-based system 
on each domain, and overall for the observation protocol. 

• An analytic engine that allows users to choose 
comparisons from among the set of MET project 
measures, including value-added on state assessments, 
value-added on supplemental assessments, teacher 
perceptions, observed classroom practice using different 
protocols, teacher knowledge for practice, and student 
perceptions. 

• An automated report generator that provides feedback 
on the predictive validity of the observation protocol as 
scored and how it performs against a set of expert-scored 
benchmark instruments. Reports can include information 
about: 


o The relationship between the scores on the 
observational protocol and the student 
achievement gains, 

o A comparison between those scores and the 
scores given by the expert raters on several 
established observation protocols, and 
o The variability in ratings between different raters 
observing the same video. 

To use the validation engine, the state or district must first 
identify a group of classroom observers. Those observers will 
watch a set of videos and rate the instructional practices 
based on their observation protocol. The state or district will 
then receive a report with information about the validity of 
their observation protocol. Results will provide information on 
individual indicators that can be used to revise the expectation 
for evidence, or possibly eliminate the indicator if it is not 
correlated with improvement in student achievement. Multiple 
observation protocols can be tested and retested regularly to 
demonstrate validity. 

About Empirical Education 

Empirical Education Inc., a research company based in Palo 
Alto, CA, specializes in studies of the effectiveness of school 
products, programs and policies. Its clients include federal, 
state and local education agencies, publishers and technology 
vendors. With the goal of making scientific evidence routinely 
available to inform practitioner decisions, the company has 
developed a suite of technology tools to increase the 
efficiency of research operations. The company has worked in 
thousands of classrooms in all regions of the U.S. For more 
information about Empirical Education and its role in the MET 
project, contact Kylene Shen at 
Kshen@empiricaleducation.com or visit 
http://www.empiricaleducation.com/. 

About the Measures of Effective 
Teaching Project 

The Measures of Effective Teaching (MET) project seeks to 
develop an array of measures that will be viewed by teachers, 
unions, administrators and policymakers as reliable and 
credible indicators of effective teaching. By determining 
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exactly what measures predict the biggest student 
achievement gains, the MET project will give teachers the 
feedback (including exemplary practices) they need to 
improve. In addition, a greater understanding about which 
teaching practices, skills and knowledge positively impact 
student learning will allow states and districts to develop 
teacher evaluation systems that will help strengthen all 
aspects of teaching — from recruitment through retention. 

The MET project has enrolled over 3,000 teachers from a 
number of school districts around the country and is gathering 
a variety of data, including videotaped teacher observations, 
student surveys, teacher surveys and supplemental student 
assessments, and represents a real opportunity for teachers 
to inform the national discussion on education reform, in order 


to determine which measures are most strongly correlated 
with high levels of student achievement. The MET project’s 
final findings will be shared broadly at the project’s conclusion 
in winter 201 1-2012. 

For more information about the MET project, please visit 
www. MET proiect.org or send an email to 
info@MET proiect.org . 

Note: Selection of a given academic, non-profit or for-profit 
organization to participate in the MET project does not 
constitute an endorsement by the MET project of that 
organization. Other organizations may exist who do work that 
is similar or equivalent to the work done by the organizations 
participating in the MET project. 



